{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8d6b6cf9",
   "metadata": {},
   "source": [
    "# Tool Input Accuracy Evaluator\n",
    "\n",
    "### Getting Started\n",
    "This sample demonstrates how to use tool input accuracy evaluator on agent data. The supported input formats include:\n",
    "- simple data such as strings and `dict` describing tool calls;\n",
    "- user-agent conversations in the form of list of agent messages. \n",
    "\n",
    "Before you begin:\n",
    "```bash\n",
    "pip install azure-ai-evaluation\n",
    "```\n",
    "Set these environment variables with your own values:\n",
    "1) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
    "2) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
    "3) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
    "4) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74ac1b5c",
   "metadata": {},
   "source": [
    "The Tool Input Accuracy evaluator performs a strict binary evaluation (PASS/FAIL) of parameters passed to tool calls. It ensures that ALL parameters meet ALL criteria:\n",
    "\n",
    "- Parameter grounding: All parameters must be derived from conversation history/query\n",
    "- Type compliance: All parameters must match exact types specified in tool definitions\n",
    "- Format compliance: All parameters must follow exact format and structure requirements\n",
    "- Completeness: All required parameters must be provided\n",
    "- No unexpected parameters: Only defined parameters are allowed\n",
    "\n",
    "The evaluator uses strict binary evaluation:\n",
    "\n",
    "    - 1: Only when ALL criteria are satisfied perfectly for ALL parameters\n",
    "    - 0: When ANY criterion fails for ANY parameter\n",
    "\n",
    "This evaluation focuses on ensuring tool call parameters are completely correct without any tolerance for partial correctness."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17346632",
   "metadata": {},
   "source": [
    "Tool Input Accuracy requires following input:\n",
    "- Query - This can be a single query or a list of messages(conversation history with agent). The original task request from the user.\n",
    "- Response - Response from Agent (or any GenAI App). This can be a single text response or a list of messages generated as part of Agent Response. The evaluator extracts tool calls from the response.\n",
    "- Tool Definitions - Tool(s) definition used by Agent to answer the query. Required to validate parameter types and structures.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d78194ee",
   "metadata": {},
   "source": [
    "### Initialize Tool Input Accuracy Evaluator\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1663a6b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azure.ai.evaluation._evaluators._tool_input_accuracy import _ToolInputAccuracyEvaluator\n",
    "from azure.ai.evaluation import AzureOpenAIModelConfiguration\n",
    "from pprint import pprint\n",
    "\n",
    "model_config = AzureOpenAIModelConfiguration(\n",
    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
    ")\n",
    "\n",
    "\n",
    "tool_input_accuracy = _ToolInputAccuracyEvaluator(model_config=model_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f92e8378",
   "metadata": {},
   "source": [
    "### Samples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba5f6441",
   "metadata": {},
   "source": [
    "#### Evaluating Correct Tool Input Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "486057ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How is the weather in Seattle?\"\n",
    "response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_weather_123\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"Seattle\"},\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "            \"required\": [\"location\"]\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "result = tool_input_accuracy(query=query, response=response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80fc6e82",
   "metadata": {},
   "source": [
    "#### Response as String (str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "626462d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Check the weather in Miami\"\n",
    "\n",
    "# Response as a simple string containing tool call information\n",
    "response_str = \"\"\"I'll check the weather for you. Calling fetch_weather with location=\"Miami\".\"\"\"\n",
    "\n",
    "tool_definition = {\n",
    "    \"name\": \"fetch_weather\",\n",
    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
    "    \"parameters\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        \"required\": [\"location\"]\n",
    "    },\n",
    "}\n",
    "\n",
    "result = tool_input_accuracy(query=query, response=response_str, tool_definitions=tool_definition)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9275f1c",
   "metadata": {},
   "source": [
    "#### Response as a List and Tool Definition as Single Dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91e1e4d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What's the temperature in Boston?\"\n",
    "response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_boston_weather\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"Boston\"},\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_definition_dict = {\n",
    "    \"name\": \"fetch_weather\",\n",
    "    \"description\": \"Fetches the weather information for the specified location.\",\n",
    "    \"parameters\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        \"required\": [\"location\"]\n",
    "    },\n",
    "}\n",
    "\n",
    "result = tool_input_accuracy(query=query, response=response, tool_definitions=tool_definition_dict)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c351c61a",
   "metadata": {},
   "source": [
    "#### Complex Tool Parameters with Multiple Fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b1147ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Can you send an email to john@example.com with the subject 'Weather Update' and tell him the weather in Seattle is 15°C and partly cloudy?\"\n",
    "response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_email_456\",\n",
    "                \"name\": \"send_email\",\n",
    "                \"arguments\": {\n",
    "                    \"recipient\": \"john@example.com\",\n",
    "                    \"subject\": \"Weather Update\",\n",
    "                    \"body\": \"The weather in Seattle is 15°C and partly cloudy.\"\n",
    "                },\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "            \"required\": [\"recipient\", \"subject\", \"body\"]\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "result = tool_input_accuracy(query=query, response=response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b274da2",
   "metadata": {},
   "source": [
    "#### Query as Conversation History (List of Messages)\n",
    "The evaluator also supports query as a list of messages representing conversation history. This helps validate parameters are grounded in the conversation context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a90e8827",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Query as conversation history instead of a single string\n",
    "query_as_conversation = [\n",
    "    {\n",
    "        \"role\": \"system\",\n",
    "        \"content\": \"You are a helpful assistant that can fetch weather information and send emails.\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\", \n",
    "        \"content\": \"Hi, can you check the weather in Seattle for me?\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": \"Please send the results to admin@company.com with the subject 'Daily Weather Report'.\"\n",
    "    }\n",
    "]\n",
    "\n",
    "response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_weather_123\",\n",
    "                \"name\": \"fetch_weather\", \n",
    "                \"arguments\": {\"location\": \"Seattle\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_email_456\",\n",
    "                \"name\": \"send_email\",\n",
    "                \"arguments\": {\n",
    "                    \"recipient\": \"admin@company.com\",\n",
    "                    \"subject\": \"Daily Weather Report\",\n",
    "                    \"body\": \"Weather information for Seattle as requested.\"\n",
    "                },\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "            \"required\": [\"location\"]\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "            \"required\": [\"recipient\", \"subject\", \"body\"]\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "result = tool_input_accuracy(query=query_as_conversation, response=response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14c41e90",
   "metadata": {},
   "source": [
    "#### Example of Incorrect Tool Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59810a22",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How is the weather in Seattle?\"\n",
    "# Missing required parameter or wrong type\n",
    "incorrect_response = [\n",
    "    {\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_weather_123\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"city\": \"Seattle\"},  # Wrong parameter name (should be \"location\")\n",
    "            }\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "            \"required\": [\"location\"]\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "# This should score 0 due to incorrect parameter name\n",
    "result = tool_input_accuracy(query=query, response=incorrect_response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_agent_evaluator_prp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
